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ABSTRACT 

Flying in poor visibility conditions, such as rain, snow, fog or haze, is inherently dangerous. However these 
conditions can occur at nearly any location, so inevitably pilots must successfully navigate through them. At 
NASA Langley Research Center (LaRC), under support of the Aviation Safety and Security Program Office 
and the Systems Engineering Directorate, we are developing an Enhanced Vision System (EVS) that combines 
image enhancement and synthetic vision elements to assist pilots flying through adverse weather conditions. This 
system uses a combination of forward-looking infrared and visible sensors for data acquisition. A core function 
of the system is to enhance and fuse the sensor data in order to increase the information content and quality 
of the captured imagery. These operations must be performed in real-time for the pilot to use while flying. For 
image enhancement, we are using the LaRC patented Retinex algorithm since it performs exceptionally well for 
improving low-contrast range imagery typically seen during poor visibility poor visibility conditions. In general, 
real-time operation of the Retinex requires specialized hardware. To date, we have successfully implemented a 
single-sensor real-time version of the Retinex on several different Digital Signal Processor (DSP) platforms. In 
this paper we give an overview of the EVS and its performance requirements for real-time enhancement and 
fusion and we discuss our current real-time Retinex implementations on DSPs. 
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1. INTRODUCTION 

In recent years the NASA Langley Research Center (LaRC) Aviation Safety and Security Program Office and 
Systems Engineering Directorate have been initiating and supporting several new projects to improve air travel 
safety. One such effort is the Synthetic Vision Systems (SVS) project where innovative methods to eliminate 
aircraft accidents caused by inadequate vision or visual cues are being investigated. 1 To augment the SVS 
project, an Enhanced Vision System (EVS) is used to provide real-time feedback from a suite of sensors to a pilot. 
The sensors can provide more information about the environment than can be obtained from direct observation, 
especially during poor visibility conditions such as rain, snow, fog or haze. The EVS uses a combination of two 
infrared sensors and a visible sensor for data acquisition (see Figure 1). A primary function of the system is to 
enhance and fuse the sensor data to increase the information content and quality of the captured imagery, and 
to present the multiple sensor outputs to the pilot on a single display. 

To perform multi-sensor image enhancement and fusion we are using the LaRC patented Retinex algorithm. 
The Retinex performs a non-linear spatial/spectral transform that synthesizes strong local contrast enhancement 
and color constancy. It performs extremely well in improving the low-contrast imagery typical of poor visibility 
conditions. However, the algorithm is computationally complex. Additionally, a processed output image rate 
of at least 15 frames per second (fps) is required to provide a flicker- free display to the pilot. To meet this level 
of performance requires specialized hardware. We are currently using Digital Signal Processors (DSPs) to serve 
this purpose. To date, we have successfully implemented a single scale, monochromatic real-time version of the 
Retinex on three different Texas Instruments (TI) DSP platforms. In the remainder of this paper we give an 
overview of the EVS and its performance requirements for real-time enhancement and fusion, and describe the 
data acquisition hardware and interfaces. We also discuss our current real-time Retinex implementations on 
DSPs and our plans to process the multiple data streams of the EVS. 




Figure 1 . The EVS pod showing the LWIR, SWIR, and Visible-band cameras, and the enclosure shell. Inaccurate 
bore-sighting can cause image registration problems. 


2. ENHANCED VISION SYSTEM 

The EVS is being developed to provide enhanced images of the flight environment to assist pilots flying in poor 
visibility conditions. The system has a long-wave infrared (LWIR), a short-wave infrared (SWIR), and a visible- 
band camera, all mounted in a pod that is flown beneath a NASA 757 aircraft. 2 Figure 1 shows the cameras 
and other components mounted in the pod, and Figure 2 shows the pod installed on the 757 aircraft. The LWIR 
and the SWIR imagers are the primary cameras for current aviation safety research efforts. The visible band 
camera will not be processed during scheduled flight tests, but remains in this discussion for future reference. 
The LWIR is a Lockheed Sanders LTC500 thermal imager and senses radiation in the 7.5-14 /jm band. It can 
image background scenery, terrain features and obstacles at night and in other low visibility conditions. The 
SWIR is a Merlin Near-Infrared (NIR) camera that senses in the 0.9-1.68 gm region and is optimal for detecting 
peak radiance from runway and taxi way lights even in poor visibility conditions. The visible-band camera is a 
Bowtech BP-L3C-II CCD that detects the 0.4-0.78 gm band and covers imaging runway markings, skyline and 
city lights in good visibility conditions. A frame from each of the three video streams generated by the cameras 
in clear weather conditions is shown in Figure 3. 

The image processing architecture for the EVS is outlined in the top of Figure 4. The digital RS422 outputs 
of the SWIR and LWIR cameras are modulated by LyteLink modules onto an optical fiber. The optical signals 
are sent from the EVS pod mounted beneath the 757, and demodulated by another pair of LyteLink modules. 
The LyteLink modules, and other processing components, are placed in a rack-mounted chassis that is situated 
in a pallet within the 757 that is approximately 120 feet away from the EVS pod. The fiber optic communications 
provides sufficient bandwidth and noise immunity to ensure clear signal propagation. The receiving LyteLink 
modules output an electrical RS422 signal that is received by a digital video distribution module and sent to 
the specialized processing board(s). 



Figure 2. EVS pod mounted forward-looking underneath the NASA Aries 757. A duplicate of the EVS is also mounted 
on a NASA van and used for ground tests of the system. 
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Figure 3. Examples of the imagery generated by each camera in good weather conditions. The images from cameras 
must be registered, enhanced, fused and displayed to the pilot in real-time. 







Enhanced Vision System Image Processing 




Figure 4. Image processing architecture and functions of the EVS. LWIR and SWIR processing can occur simultaneously 
in a multiprocessor system. The SWIR data is used as the baseline for registration since it has the smallest held of view. 


The functions performed by the processing board are shown in the bottom of Figure 4. The multi-spectral 
data streams from the cameras must be registered, enhanced and fused into a single image stream. The data 
streams from the cameras must be registered to remove field of view (FOV) and spatial resolution differences 
between the cameras, and to correct bore-sighting inaccuracies. Table 1 gives the relevant characteristics of 
the sensors. Registration of the LWIR data to the SWIR is performed by applying an affine transform to the 
LWIR imagery. 3 The SWIR channel is selected as the baseline because the optics of the SWIR camera have 
the smallest FOV. Registration is performed by first selecting a set of control points based on corresponding 
features. The control points are analyzed using multiple linear regression to approximate the coefficients of the 
affine transform. The transformed image is then resampled using bilinear interpolation to align the registered 
image data on the same grid as a reference SWIR image. The same transform can then be used on all other 
video frames since both the optical parameters and the camera alignment should not change during flight. 



SWIR 

LWIR 

CCD 

Image Dimensions (pixels) 

320H x 240V 

320 H x 240V 

542LT x 497V 

Optics FOV 

34 ° H x 25°V 

39 °H x 29° V 

34° H x 25°V 

Detector Readout Frame Rate 

60 Hz 

60 Hz 

30 Hz (interlaced) 


Table 1: Sensor Specifications 

Enhancement is performed to improve the information content of the captured imagery particularly during 
poor visibility conditions. For enhancement, we are using the LaRC patented Retinex algorithm. The Retinex 
is a general-purpose image enhancement algorithm that is used to produce good visual representations of scenes. 







It provides simultaneous dynamic range compression, color constancy, and color and lightness rendition. 4, 5 The 
Retinex provides an ideal solution in the context of the EVS because of its superb performance for improving 
low-contrast images. A visual servo concept has been created that generalizes the Retinex and thus optimizes 
the processing for turbid images. 6, 7 The Retinex can also be used as a fusion engine since the algorithm 
performs nearly symmetric processing on multi-spectral data, and applies multiple scaling operations on each 
spectral band. Multiple camera inputs can be distributed onto these multi-spectral/scale processing chains and 
fused using the weighting and summation properties of the Retinex. 8 The fused data stream, output as a 
standard NTSC signal, is fed into a low-resolution heads-up or heads-down display for viewing by the pilot. 
The fused video stream contains more information than any individual spectral band and also provides the 
additional benefit of producing a single output for the pilot to observe instead of multiple video sources. 

Several EVS parameters are used to provide a set of baseline requirements and constraints for real-time 
Retinex processing. First, our initial performance goal is to achieve a display rate of 15 fps, instead of the 
de facto standard of 30 fps for real-time video. We can use this reduced rate because several human factor 
studies have shown that an update rate of 15 fps is more than sufficient to avoid flicker, accurately portray 
motion, 9 and not cause pilot-induced-oscillations (PIOs). 10-12 PIOs are unstable flying states that can occur 
when a pilot views and reacts to an instrument or display that is updated too slowly (< 12 fps) 13, 14 Second, 
the cockpit displays are low-resolution (320 x 240). This significantly reduces the amount of image data that 
must be processed. Additionally, if a processed frame size of 256 x 256 is used, only 20 percent of the horizontal 
component of the image would be lost. Third, the SWIR and the LWIR are the only two cameras currently 
being processed and both of these cameras are monochromatic. This reduces the number of spectral bands that 
have to be processed from potentially 9 (3 per sensor) to 2 (1 for each of the LWIR and SWIR cameras). Fourth, 
the space allocated for image processing within the EVS will hold at least a single PCI board thus allowing a 
board-level (vs. chip-level) solution for processing. Even with these reduced processing requirements, execution 
of the registration, enhancement, and fusion operations in real-time (15 fps) requires specialized hardware. We 
are currently using DSPs to meet the processing requirements and are focused on implementing a real-time 
version of the Retinex since it is the primary, and most computationally expensive, image processing function 
within the EVS. 


3. DIGITAL SIGNAL PROCESSORS 

We are currently using three Texas Instruments (TI) DSPs for implementation and performance evaluation 
of the real-time Retinex. TI processors were chosen because of their flexible and powerful architecture, good 
support tools, availability of the DSPs to the researchers, low cost of evaluation boards, and our past familiarity 
with using TI processors. Many other DSPs, such as Analog Devices SHARC processors, would also provide 
reasonable hardware solutions for implementation. All of the state-of-the-art TI DSPs that were chosen are 
based on an advanced very-long-instruction-word (VLIW) 15 architecture. This type of architecture achieves 
high performance by exploiting instruction-level parallelism. Multiple execution units operate in parallel to 
execute multiple instructions during a single clock cycle. 

The three TI DSPs we are using are the TMS320C6711 (C6711), the TMS320C6713 (C6713), and the 
TMS320DM642 (DM642). The C6711 is a 32-bit floating-point processor that performs up to 1200 million 
instructions per second (MIPS)/900 million floating point operations per second (MFLOPs) at a clock rate of 
150 MHz (6.67 ns instruction cycle time). The C6713 is also 32-bit floating-point processor and it performs up 
to 1800 MIPS/ 1350 MFLOPS at a clock rate of 225 MHz (4.4 ns instruction cycle time). The architecture of 
the C6713 is very similar to the C6711, and code operating on the one device directly ports over to the other. 
The TMS320DM642 (DM642) is based on the same advanced VLIW architecture as the C6711 and C6713 but 
it is structured differently than the other two processors. The DM642 is a 32-bit fixed - point processor that 
provides up to 4800 MIPS at a clock rate of 600 MHz (1.67 ns instruction cycle time). Figure 5 shows each 
processor on an evaluation board. To test the system, video data is fed into the boards from a standard NTSC 
camera and the processed output is displayed on a standard monitor. 

The floating-point architectures of the C6711 and C6713 allow calculations to be performed using a 24-bit 
mantissa and an 8-bit exponent thus yielding much greater precision and dynamic range than 32-bit fixed-point 




Figure 5. Test setup. Video is captured from the NTSC camera, processed on one of the DM642, C6713 or C6711 DSP 
boards, and displayed on a monitor. The C6711 board is shown with a daughtercard that is used for video encoding and 
decoding. 


representation. Double-width precision calculations using 64 bits (with 53-bit mantissa and 11-bit exponent) 
can also be performed on the C6711/13 at the sacrifice of speed since multiple cycles are used for each operation. 
However, in general the extra precision and accuracy offered by these floating-point architectures is not required 
for EVS processing thus, the limited dynamic range (2 31 — 1) and numeric accuracy of (0.5 x 2 -32 assuming 
Q0.31 format) of the fixed-point DM642 are sufficient. The DM642 also offers additional features that improve 
its performance when processing image data, such as new instruction extensions and media ports built directly 
onto the chip. 16 The media ports include three configurable video ports that can support either video capture 
and/or video display modes. 


4. CURRENT RESULTS 

We tested our first solution for the real-time, single-scale, monochromatic version of the Retinex algorithm on 
the C6711 DSP and attained a processing rate of 20.7 fps. 17 This frame rate would only be sufficient to enhance 
one channel of EVS data. To improve and compare performance we mapped the same code that executes on 
the C6711 onto the C6713 and achieved a processing rate of 28 fps. 18 This value is less than the expected 
linear increase to 31 fps due to the slower external memory interface (EMIF) clock used for the SDRAMs of 
the C6713. 

Next, we ported the Retinex algorithm to the DM642 platform. The DM642 uses different image capture 
and display drivers, direct memory access (DMA) transfer buffers, and fast Fourier transform (FFT) algorithms 
than the C6711, but the core of the algorithm remains the same. Our initial results were limited to 34 fps. 18 
Recently, after correcting erroneous code generated by the compiler, using hand-coded optimizations such as 
intrinsics, and rewriting the Retinex code to exploit the architectural advantages of the DM642, we have achieved 


a processing rate of 92 fps. This rate is more than sufficient to perform the required registration, dual channel 
Retinex enhancement, and fusion algorithms within the EVS. 

5. CONCLUSIONS AND PLANS 

The EVS will provide a significant improvement in the information provided to a pilot about the flight en- 
vironment, particularly during times of poor visibility. A flight test is scheduled for mid-2005 on the NASA 
LaRC 757. During this test digital video will be acquired from the SWIR and LWIR cameras. These video 
streams will be registered, Retinex enhanced, fused, and displayed for analysis. Hardware implementation of 
these functions will be based on DSP platforms because of the computational complexity of the algorithms. 
We have successfully implemented the core real-time Retinex algorithm operating at 20, 28 and 92 fps on a 
TI C6711, C6713, and a DM642 DSPs, respectively. The DM642, 92 fps implementation should provide the 
necessary bandwidth to process multiple spectral bands, and to add the registration and fusion operations to 
the system. 

The major challenges that remain to complete the system are to (1) modify the inputs into the DM642 
processing board to accept the digital RS422 signals from the EVS cameras, (2) modify the algorithm to process 
12-bit parallel digital pixel data instead of the current 8-bit analog data, and (3) integrate the registration 
and fusion algorithms. Changing from analog to digital may involve developing a custom converter board or 
migrating to a different processing platform that has the appropriate interface. Processing 12-bit data will 
primarily affect the scaling operations used within the fixed-point DM642 and will also increase the amount of 
memory required to store image data. These issues are currently under study. 
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